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ABSTB ACT 

Evaluation has generally not been accountab 
terms of its promoting a «net-benefit . " The tern "net-benef 
than "benefit" is used because a given amount of legitimate 
nay come at the expense of an inordinate expenditure of eva 
resources cr energy- If any aspect of an evaluaticn is unli 
provide any net-benefit to humanity as far as the overall e 
is ccncerned, then it probably shouldn't be done given the 
scarcity of evaluation resources and energy. Examples of 
E€t-nonbenef icial energy- wasting evaluation activities iaci 
carrying ctJt cverly complex statistical analyses; (2) disse 
aad use (cr lack thereof) of the draft version of the Stand 
prcdcced by the Committee on Standards for Educational Evai 
which forbade the draft version to be cited, duplicated, or 
distributed without written permission of the Chairman of t 
Standards Committee; and (3> the publication of articles in 
excess spending of time in adhering tc style guidelines- De 
whether an evaluation activity has the potential to lead to 
net-benefit is cl-aarly not always an easy task, but it is a 
toward achieving accountability- (BH 
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Introduction 

Today, many people in education would readily agree that educational 
evaluation is a ^^e^uine profession. Advanced degrees in evaluation are 
offered at many universities, semi-official standards have been published, 
and many evaluators certainly charge enough for their services. Unfortunately, 
as in any profession, many members have lc;5t -.ight of the rationale for their 
existence. That rationale is not to keep progj^m developers and coordinators 
honest and in line. It is not to display academic prowess in developing 
evaluation desigriS or using statistics or testing. It is not to provide 
information to decision makers who then use the data to promote their own 
self-interest concerns. 

Purported Goals of Evali;iation 

Scriven (1967) would have us believe that the goal of evaluation is to 
ascertain merit. On the other hand, Stufflebeam at al. (1971) say that 
providing information for judging decision alternatives is why evaluation 
takes place. Others such as Parlett and Hamilton (1977) would settle for 
evaluation as enlightenment whereby the level of the discussion is raised. 

Notice that none of these references to what the goal of evaluation is 
mentions directly any net benefit to the recipient of the program or product. 
What good is the ascertainment of merit i£ no one benefits as a result? As 
for decision maldng wrlth the use of evaluative data, can an e\'aluation have 
been worthwhile even if no one benefits as a result of a decision made? 
Although enlightenment may be considered a benefit to some, this approach 
implies that the professionals involved are the Tpain focus of the benefit- 
providing potential of the evaluation. Granted that the determination of 
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merit, the decision making, and the enlightenment are valued because of their 
being hopefully conducive to the subsequent production of beneficial effects, 
nonetheless, evaluation has generally not been accountable terms of its 
promoting a net benefit - 

Net Benefit as a Goal of Evaluation 

The term "net-benefit" rather than "benefit" has been used becaui • a 
given amount of legitimate benefit may come at the expense of an inordinate 
expenditure of evaluation resources or energy. In a simplistic sense, such 
an evaluation or e\'aluative activity would not have been worth the effort. 
Notice that Scriven's concept that evaluation can and should be cost-free 
(he means, of course, net cost-free) does not go far enough. An evaluation 
that was cost-free only in the sense that it saved more money than it cost 
would have been worthwhile only if that money saved had then been used to 
provide a worthwhile benefit. 

Within an evaluation, the net-benefit criterion should also be applied. 
If any aspect of an evaluation is unlikely to provide any net-benefit as far 
as the overall evaluation is concerned, then it probably shouldn't be done 
given the relative scarcity of evaluation resources and energy. 

Despite widespread fiscal austerity, evaluators have not generally given 
credence to the evaluation energy crisis that exists. Much of the energy 
spent by evaluators does not residt in a net benefit to anybody, except 
perhaps to the evaluator who is paid for services rendered. Evaluation does 
not yet have the luxury, that basic research perhaps has, of justifying a 
major part of its existence on the possibility of future, unknown uses of its 
current findings. 
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House (1976) touched upon the subject in his treatise on justice in 
evaluation. In his discussion, however, he focused on the overall effects of 
evalxiation on a particular population. In an example cited in his article, 
an evaluation activity that would violate an individual's self-esteem or 
would leave a respondent vulnerable would be deleted even if the evaluation 
design suffered from a technical viewpoint. In a definite sense, it was 
rc :her easy for House to recommend that an activity that resulted in detriment 
t.: an individual be dropped from an evaluation. 

It is contended that another step must be taken; namely, the approval ox 
disapproval of evaluation activities must take into account the net benefit 
of the decision. Activities that are not likely to result in net benefit to 
humanity should probably be dropped even though no negative result:^ (other 
than wasted energy) iray occiir through their use. Activities that are likely 
to result in net benefit should be carried out if possible. 

Examples of Net Non-Beneficial. Energy-W asting Evaluation Activities 

One frequent energy waster is the activity of carrying out overly complex 
statistical analyses. In many evaluations, the use of niultiple regression 
with exact probabilities resulted in no benefit to anyone. Yet it used up a 
substantial amount of evaluation energy in the form of computer use, personnel 
time (for planning, writing and reading) , and paper (computer as well as draft 
and final report). And yet as Bryk (1978, P. 40) has noted: "And despite the 
fact that our computer technology permits incredible complex multivariate 
analyses of such data in the indivLdiial context it cannot rival f - integrative 
capacity of the clinical mind." 
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Baker (I976) advocated a T^ap date, approach to evaluation. Ker 

colleagues at the C^^^^^ for the Study of Evaluation at UCLA, however, in 

the official CSE ni^^^^^P^- series (in particular, numbers 4 and 5) do not 

appear to be disci-p^^^ precursors by any means. A major argument made 

by those advoc^tin^ ^^^^ approach is that i- is better to do a good 

job on 3 few criti^^^ variables than to do a shallc . job on a comprehensive 

set of variables. important, however, to go one step further and argue 

that a comprehensi^^ variables should in general not be investigated 

even if in-depth a^^^^^^es were feasible and affordable. Only if such 

analyses are lilcel/ result in some net benefit should they be carried out. 

Otherwise, v^i^^hl^ evaluation energy is being wasted. 

A negative exaJ^^^^ national level was prodursd by the Committee 

cn Standards fo^ gV^l^tion. The Standards themselves are not seen as a waste 

of evaluation e^er^* however, the method of dissemination and use (or lack 

thereof) of the dr^^^ version was an exainple of non-net-beneficial thinking. 

The dra^t copy da^^^^ November, 1978, on every page had the following warning: 

dur>iTcs'J"'^0'^ distribution withov.t the written Demission 
''Not for citation, r'--^*=— . ^ 

. ^, ^ . - the Joint Committee on Standards for Educational Evaluation/' 

of the Cnairmaji ot 

On a particular oc^^°^ ^^^^ permission was requested in order that the 

Standards be (1) ^ metaevaluate an isolated Title IV project and (2] 

reproduced foj* ^igi^^^ution to advanced graduate students in an NIE-sponsored 

^ ^^i"^ was denied, with the main reason cited being that of the 
seminar, tne j^^ciue-^^ ' 

confusion that tnigl'^^ ^"^^^ ^'^^^^ ^ document were used, and subsequently 

, . out. Although the denial of the Committee was honored, 

a rinai version cai^ ^ 

_.^ed: Which situation would be of more (net) benefit? 
a question wa^ rai^ 
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Situation 



1. Draft version of Stand- 
ards not citable or 
dlstribut ab 1 e 



Benefit 

No future confusion 
when final form out 



2. Draft version of Stand- 
ards citable and distribu- 



Detriment 

Project evaluation 
denied use of a re- 
spectable version of 
the Standards- - 
Grad'iate students not 
allowed to obtain copy 
of Draft Standards 

Possible future con- 
fusion over which form 



Project metaevaluated 

with draft of Stand- 

tabfe ards ; Graduate students of Standards is being 

have copy of draft of cited 
Standards to serve as 
guide 

It is argued that the use of the draft version of the Standards could 
have been of enough net benefit to consumers that the denial o£ permission 
to cite or reproduce them was unwarranted. 

Another example of evaluation energy waste comes from the publication 
field. Because of established styles and some editors' hangiq)s, a lot of 
energy is wasted in the process of getting articles published. The importance 
of style guidelines such as provided by the Publication Manual (1974) of the 
American Psychological Association .is recognized; however, strict adherence 
for some writers may require an inordinate amount of time which could be 
better spent doing something else. When listing referencer, for example, is 
it really of net benefit to have the author spend time on learning (and 
implementing) the fact that (1) a comma must be placed before the ampersand 
even though there are only two co-authors, (2) a single space must occur after 
each period following an author's first name initials (no full names, pi easel), 
a^id (3) the rules for capitalisation within an article title listed in the 
references section are different from those for journal names? Again, a 
caution is in order. The following of an established (and, hopefully, empirically 



EKLC 



Morris K. Lai 



sound) style is not being criticized. What is being challenged is the 
practice of adhering to a style even to the detriment of the allocation of 
evaliaation energy. More generally each of the rituals of publishing is 
being challenged to meet the following criterion: Does it have the potential 
to result in a net benefit to humanity--or could the time and energy be b.Jtter 
spent in another potentially more beneficial activity? 

It is, of course, much easier to advocate tha:. to carry out a net-beneficial, 
energy- efficient evaluation. ld<tas on how to conserve evaluation energy or 
OToduce net-benefit will not be discussed in depth in this presentation. 
Promising areas/concepts include rules of thumb (Lai, 1979), lean daca (Baker, 
1976), styleless journals, efficiency/beneficial checklists (analogous to 
Scriven's [1974]), and recognition of the paper shortage. 



Educational Importance 

The conservation of evaluation energy must itself be assessed against 
the criterion of net potential benefit. If it is assumed :hat evaiuxition can 
have a beneficial effect, th the efficient use of evaluation ener^ allows 
more things to be evaluated within the existing evaluation energy supply. A 
stronger argiment, however, is needed. 

Despite Parkinson *s L--:^, which might suggest that evaluation would simply 
expand to consume any extra energy made available, it would appear legitimate 
to argue that saving evaluation energy would enable more productivity (perhaps 
ia non-evaluative areas) and thus result in increased benefits to various 
parties. Not only woul "uators be wasting less time and energy, rut 
audiences of evaluaticn Zs wouli also waste less time learning about 

evaluation activities that benefit no one. 
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Conclusion 

Determining whether an evaliiation activity has the porential to lead to 
net benefit is clearly not always an easy task; however, it is not any 
easier to determine whether an evaluation has met accuracy, utility, propriety, 
and feasibility standards. A net -beneficial , energy- conserving approach to 
evaluation emphasizes that evaluation indeed is a niost important human 
activity. At the same time,, such an approach reminds us that evaluation 
should no*c be a game in which awards are made on the basis of outward 
appeaxsnce. Many extremely "proper" evaluations are actually energy-wasters 
and non-beneficial. They should be created accordingly. 
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